Calculation method of line-of-sight direction based on analysis and match of iris contour in human eye image

ABSTRACT

The invention provides a calculation method of line-of-sight direction based on analysis and match of iris contour in human eye image, including: a data driven method, for stable calculation of 3D line-of-sight direction via inputting human eye image to be matched with synthetic data of virtual eyeball appearance; two novel optimization matching criterions of eyeball appearance, which effectively reduce effects of uncontrollable factors, such as image scaling and noise on results; a joint optimization method, for the case of continuously shooting multiple human eye images, to further improve calculation accuracy. One application of the invention is virtual reality and human computer interaction which is under the principle that shooting eye images of a user and calculating line-of-sight direction of user to enable interaction with intelligent system interface or virtual realistic object. The invention can be widely used in training, games and entertainment, video surveillance, medical care and other fields.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to Chinese Patent Application No.201610218355.7, filed on Apr. 9, 2016 and entitled “CALCULATION METHODOF LINE-OF-SIGHT DIRECTION BASED ON ANALYSIS AND MATCH OF IRIS CONTOURIN HUMAN EYE IMAGE”, which is hereby incorporated by reference in itsentirety.

TECHNICAL FIELD

The present invention relates to the field of computer vision and imageprocessing, and in particular, is a calculation method of line-of-sightdirection based on analysis and match of iris contour in human eyeimage.

BACKGROUND

Line-of-sight tracking/eye-movement tracking holds great significancefor the understanding of user behavior and efficient human-computerinteraction. More than 80% of information perceptible to human isreceived by the human eye, of which more than 90% is processed by thevisual system. Therefore, the line-of-sight sheds great light onreflecting the interaction process between human and the outside world.In recent years, value of the line-of-sight tracking technology inapplication gradually stand out, thanks to the rapid development ofvirtual reality technology and human-computer interaction technology; onthe other hand, calculation of the line-of-sight direction remains agreat challenge in the field of computer vision. Up until now, thesolution has mostly been based on active light source and infraredcamera, which requires additional hardware, and demanding conditions ofthe application environment. In one alternative method, a single camerais employed for shooting a human eye image prior to calculation of theline-of-sight direction, eliminating the need for assuming the activeillumination, however, a large number of training samples are requiredto be obtained in advance, for the sake of conducting learning andderiving a regression calculation model. For example, an early-stageneural network system, proposed by Baluja and Pomerleau, requires theuse of thousands of training samples for training. Tan et al., proposeda method based on local linear interpolation, in which the eye image andcoordinates of the line-of-sight undergo mapping, a system which needsabout 200 training samples.

In order to reduce the demand on the number of training samples,Williams et al. proposed a semi-supervised method capable ofsimultaneously utilizing both labeled samples and unlabeled ones fortraining. Lu et al. proposed a self-adaptive regression framework basedon sparse optimization, which allows the use of fewer training samplesfor calculation, and is able to address a series of related issues inthe calculation of line-of-sight at the same time. Sugano et al. adopteda method to automatically generate training samples, before applyingthem to system training, via extracting visual saliency from videos. Theabove methods are disadvantageous in that, the position of the head isassumed to be fixed; and more training samples are in need to solve theproblem of head movement, if the methods are to work under the conditionin which the position of the head changes.

In order to completely avoid system training, Yamazoe et al. and Heymanet al. came up with a method to realize the calculation of line-of-sightvia calculating the position of the iris center relative to the eyeballcenter, considering that the line-of-sight direction is determinedsolely by eyeball orientation, which is obtainable by calculation of theorientation of the iris disc or the central position thereof. Theirmethod requires 3D modeling of the head, and precise tracking of 3Dfeature points of the face, including the position of an eye corner andthe central position of the eyeball. In practice, these feature pointsare usually difficult to precisely extract, sometimes even invisible.Ishikawa et al. utilized a method to track facial feature points basedon active appearance model (AAM), which also encountered the sameproblem. In some other methods, an ellipse is employed to fit the iriscontour, before the ellipse is subject to reverse projection to form acircle in 3D space. The method derives from the fact that the iriscontour can be regarded as an approximate circle, the projection ofwhich in a two-dimensional image is elliptical, and it is possible forthe orientation of the iris in a 3D world to be worked out via analysisof the elliptical shape. This method is common as one based on the shapeof the iris contour. However, the traditional iris contour analysismethod may not be reliable in practical application. What accounts forthis is that, in an image, the iris region is small in area while largein noise, rendering a precise extraction of the contour very difficult,adding to the fact that a nuanced error in a few pixels extracted fromthe contour is all that is required to cause enormous deviation in thecalculation of line-of-sight. Therefore, in many cases, the only choiceis to shoot a human eye image with ultra-high resolution, or to use awearable camera to improve the precision, which raises the requirementsfor hardware and imposes further restrictions on practical applicationscenarios. Given the aforementioned methods, the present inventionprovides a calculation method of line-of-sight direction based onanalysis and match of iris contour in human eye image, in which virtualgeneration of the appearance of the iris is combined, mainly forovercoming the problem of poor stability and low precision of thetraditional iris contour matching method, concerning a normal-resolutionhuman eye image that has been shot, so as to realize high precisioncalculation of 3D line-of-sight.

SUMMARY

According to the actual needs and technical problems mentioned above,the present invention is intended to: provide a calculation method of 3Dline-of-sight, via fitting virtual eyeball appearance, generate datasetof virtual eyeball appearance under different line-of-sight directions,and via matching with a human eye image, realize calculation of 3Dline-of-sight direction of human eye. The method has no additionalrequirement for the system, and only uses the human eye image shot by asingle camera as input. Meanwhile, the present method realizes betterrobustness, compared with other similar method, via proposing twotechnologies for analyzing and matching the shape of the iris contour.

Technical solutions of the present invention: a calculation method ofline-of-sight direction based on analysis and match of iris contour inhuman eye image, firstly, concerning obtaining the human eye image, thepresent invention includes the following processes: using a singlecamera to shoot an image including the facial region of a user; fixingon a left-eye or right-eye region using an existing face analysismethod; pre-treating the extracted human eye image to obtain an imageafter brightness correction, and generating pixels on partial iris edgevia edge detection.

Secondly, a method is invented for synthesizing virtual eyeballappearance and establishing dataset for different line-of-sightdirections: establishing a 3D sphere model for the eyeball, and addingimportant details such as iris contour thereon; traversing physicallyfeasible eyeball orientation parameters, namely, rotation angles aroundthe horizontal and vertical axes, for each of the eyeball orientation,projecting corresponding 3D virtual eyeball appearance to a 2D plane(corresponding to the direction straight ahead of the eyeball), andrecording the 2D coordinate information, such as the iris form, centralposition of the eyeball, and iris central position after the projection;and storing all the rotation angles and the correspondingly generated 2Dcoordinate information in a dataset.

Further, a method is invented to match the human eye image with thesynthetic virtual eyeball appearance, wherein via maximizing thematching degree, selecting virtual eyeball appearance which is the mostconsistent with the human eye image, obtain the corresponding eyeballorientation and position. Concerning matching of the eyeball appearance,matching algorithms for the constraints based on both the measurement ofcircular symmetry and the measurement of iris contour matching degreeare invented: regarding the former one, specifying any one group ofmatching parameters (relative translation and the eyeball orientation),and determining coordinates of the corresponding iris contour of thevirtual eyeball and the iris central position; overlapping thecoordinates onto the human eye image, conducting measurement of thevariation pattern of the pixel gradient of the human eye image in theproximity of the iris contour, studying 2D circular symmetry with theelliptic contour of the iris as reference, and using the result as acriterion to measure the matching effect; regarding the latter one,specifying any one group of matching parameters (relative translationand the eyeball orientation), and determining coordinates of thecorresponding iris contour of the virtual eyeball; traversing the pixelson the iris edge extracted from the human eye image, calculatingdistances between the edge pixels and the iris contour of the virtualeyeball; examining distribution of the distances, and counting thenumber of the distances which are significantly different from theothers, and the less the number is, the better the measurement result ofthe iris contour matching degree.

Additionally, regarding the continuously shot human eye images, on thecondition that the central position of the eyeball remains unchanged orhas been aligned, a joint optimization method is invented, and iscapable of improving the accuracy of calculating the line-of-sightdirection from multiple human eye images. On the basis of the abovematching result of the human eye image with the virtual eyeballappearance, for each human eye image, respectively calculate to obtainthe eyeball orientation and the coordinates of the eyeball centralposition; and for the result of all the images, excluding the eyeballcentral coordinates with obvious deviation, and conducting weightingcalculation of standard coordinates of the eyeball center using theremaining coordinates; individually carrying on with the appearancematching, while adding one optimization constraint, namely, the matchedeyeball central coordinates coincide as much as possible with thestandard eyeball central coordinates, namely, the matched eyeballcentral coordinates coincide as much as possible with the standardeyeball central coordinates. The calculation result is updated as thefinal result of the eyeball orientation in each human eye image.

The specific implementation steps of the present invention are asfollows:

(1) constructing a sphere eyeball model, traversing all eyeballorientations with different physical feasibilities, generating 2Dvirtual eyeball appearances with different orientations via geometriccalculation, and storing all the eyeball orientations and correspondingvirtual eyeball appearance data in a dataset, for use in specificapplications;

(2) during application, firstly shooting a facial image of a user,fixing on a left-eye or right-eye region, pre-treating the human eyeimage, completing brightness correction and extracting pixels on irisedges in the human eye image;

(3) regarding the shot and pre-treated human eye image and the virtualeyeball appearance data in the dataset, matching the human eye imagewith the virtual eyeball appearance data via a matching optimizationalgorithm of the human eye image and the virtual eyeball appearancedata, where a matching result determines orientation and position of theeyeball which best match with the human eye image; and

(4) regarding continuously shot human eye images, further conductingjoint optimization on the basis of the eyeball appearance matching instep (3), on the condition that the central position of the eyeballremains unchanged, or that the human eye images have been aligned, andprecisely and simultaneously calculating 3D line-of-sight directioncorresponding to each image.

A method for generating the virtual eyeball appearance data in step (1)is as follows: firstly, establishing a 3D sphere model of the eyeball,and adding important elements of a circular iris contour on the surfaceof the 3D sphere model; traversing eyeball orientations varying inphysical feasibilities which is different rotation angles around thehorizontal and vertical axes, for each of the rotation angles,projecting corresponding 3D virtual eyeball appearance to a 2D planewhich is to a position just in front of the corresponding eyeball, andrecording the projected 2D eyeball appearance data of iris contourcoordinates, eyeball central coordinates and iris central coordinates;and storing all the eyeball orientations and the virtual eyeballappearance data corresponding to the eyeball orientations in thedataset.

The matching optimization algorithm of the human eye image and thevirtual eyeball appearance data in step (3) is as follows: matchingparameters to be calculated are relative translation amount of the humaneye image and the virtual eyeball appearance in a 2D image domain, andthe eyeball orientation corresponding to the virtual eyeball appearance,and optimizing matching degree of the human eye image and the virtualeyeball appearance via finding the best values of the two matchingparameters, so as to realize matching of the human eye image and thevirtual eyeball appearance data.

The matching degree is calculated by the following functionalmeasurements:

(31) measurement of circular symmetry: pixels in the proximity of thevirtual eyeball iris contour in the human eye image have better circularsymmetry, when the matching tends to become ideal;

(32) measurement of iris contour matching degree: calculating distancesbetween pixels on the iris edge in the human eye image and the virtualeyeball iris contour, when the matching tends to become ideal, and thedistances tend to become equal.

A method for conducting the measurement of the circular symmetry in step(31) is as follows: coordinates of the iris contour of the virtualeyeball appearance and coordinates of the central position of the iriscorresponding to any one group of matching parameter may be determined;overlapping these coordinates onto the human eye image, via continuouslysampling values of pixels on the human eye image, along both thepositive and negative directions indicated by lines connecting the iriscenter of the virtual eyeball appearance and points on the iris contour,while taking the points as reference, so as to obtain 1D pixel columnvectors, and the sampling region is proportional to the distancesbetween the points on the iris contour of the virtual eyeball appearanceand the iris center of the virtual eyeball appearance; sampling viatraversing the points on the iris contour of the virtual eyeballappearance, and combining all the obtained 1D column vectors into a 2Dmatrix; and finally, calculating distribution consistency of each columnof numeric of the matrix or a gradient matrix of the matrix in thevertical direction, the consistency may be measured in a matrix kernelfunction, the correlation coefficient of each column, the concentrationof singular values, and the higher the consistency, the better thecircular symmetry.

A method for conducting the measurement of the iris contour matchingdegree in step (32) is as follows: regarding any one group of matchingparameters, determining coordinates of the iris contour of the virtualeyeball appearance corresponding to the matching parameters of thegroup; traversing the pixels extracted from the iris edge of the humaneye image, and calculating the distances between the edge pixels and theiris contour of the virtual eyeball appearance; and reviewingdistribution of the distances, in which the less the number of thedistances which are significantly different from the others in astatistic sense, the better the measurement result of the iris contourmatching degree.

A method for the joint optimization in step (4), and the method foraccurately calculating the 3D line-of-sight direction corresponding toeach image simultaneously are as follows: under the assumption thatcentral position of the eyeball remains unchanged or has been alignedwhen the images shot, conducting the matching of the human eye imageswith the virtual eyeball appearance in step (3), and calculating theeyeball orientation corresponding to each human eye image and thecoordinates of the central position of the eyeball; excluding thecentral coordinates of the eyeball with obvious deviation therein, andconducting weighting calculation of standard coordinates of the eyeballcenter using the remaining coordinates; and individually carrying onwith the optimization in step (3), while adding one optimal constraint,namely, coinciding the matched eyeball central coordinates as much aspossible with the standard eyeball central coordinates, and updating thecalculation results as the eyeball orientations in the human eye images,namely, the final results of the 3D line-of-sight direction.

Compared with the other method based on iris appearance analysis, thebeneficial characteristics of the present invention lie in that: (1) adata driven method is invented, and via synthesis of virtual andphysically feasible eyeball appearance data, the calculation ofline-of-sight direction is transformed into a problem of matching a realhuman eye image with the synthetic multiple orientation virtual eyeballappearance, which is conducive for a stable calculation; (2) concerningthe iris appearance matching, two novel optimization criterions areinvented, including a measurement criteria of circular symmetry and ameasurement criteria of iris contour matching degree, which differ froma traditional method in that, rather than requiring exact coincidencewith the iris contour, the matching is capable of flexibly measuring thesimilarity of the iris contour in shape, so as to efficiently reduce theeffects of uncontrollable factors, such as image scaling and noise onthe results; and (3) in the case of continuously shooting multiple humaneye images, and under the assumption that central position of theeyeball remains unchanged or has been aligned, the present inventionproposes a joint optimization method, which is capable of respectivelycalculating the line-of-sight direction more accurately from multiplehuman eye images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a relationship between an iris imageand a line-of-sight direction according to the present invention;

FIG. 2 is a diagram illustrating generation of virtual eyeballappearance according to the present invention;

FIG. 3 is a diagram illustrating the matching of a human eye image withvirtual eyeball appearance according to the present invention;

FIG. 4 is a flowchart illustrating calculation of the line-of-sightdirection based on a single human eye image according to the presentinvention;

FIG. 5 is a flow chart illustrating joint optimization of theline-of-sight direction based on multiple human eye images according tothe present invention.

DESCRIPTION OF EMBODIMENTS

Specific implementations of the present invention will be described indetail below in conjunction with the accompanying drawings.

Referring to FIG. 1, which is a diagram illustrating a relationshipbetween an iris image and a line-of-sight direction according to thepresent invention, a simple 3D sphere model is employed to establish amodel of an eyeball, in order to calculate a 3D line-of-sight directionof a human eye image shot by a camera. Specifically, the eyeball isconsidered to be a standard sphere with circular iris region on thesurface, and as to the eyeball model, the model parameters only includethe radius of the eyeball and the diameter of the iris, and is henceeasy to analyze and calculate. On such basis, line-of-sight directionmay be determined by using a ray approximately starting from the eyeballcenter and through the iris center. On the other hand, it is known froma basic law of 3D geometry that, in this model, the line-of-sightdirection is consistent with the direction of a normal vector of thecircular iris region. Therefore, the problem of calculating the 3Dline-of-sight direction can be transformed into one of calculating theorientation (the normal direction) of the iris region.

When a camera is used to shoot human eye appearance, the iris contour,which is originally circular, is rendered elliptical in the image due toperspective. It is easy to prove that, when movement of a 3D eyeballleads to change in the iris orientation, the shape of ellipticalprojection of the iris contour is directly determined by the irisorientation. For example, when the eyeball and the iris are directlyfacing a shooting direction, the shot 2D iris contour is of standardcircular shape; and if the eyeball rotates, the iris orientation rotatesalong with the eyeball, resulting in contraction of the 2D image thereofafter projection along the rotating direction, and the projected 2Dimage is presented as an ellipse contracted along its short axis.Accordingly, 3D orientation of the iris may be restored via calculatingthe shape of the elliptical iris contour (determined by the long axisand the short axis). As a result, the problem of calculating the 3Dline-of-sight direction from a human eye image may eventually betransformed into analysis and calculation of a 2D iris contour in animage.

Referring to FIG. 2, which is a diagram illustrating generation ofvirtual eyeball appearance according to the present invention, a methodis proposed to conduct calculation and drawing based on a 3D sphereeyeball model, in order to avoid generating a large amount of virtualeyeball appearance data with physical feasibility, without performingactual shooting. Firstly, a parameterization scheme is determined forgenerating virtual data. Specifically, the eyeball orientation is takenas indicators, and is further decomposed into two rotating manners ofthe eyeball around an X axis and a Y axis. Marking rotation angles ofthe two directions as u and v, respectively, and the overall rotationangle of the eyeball may be approximated as arcsin ((sin² u+sin² v)⁻²).Herein, it is taking into consideration that generation of the virtualeyeball appearance shall guarantee the physical feasibility, therefore,a constraint is imposed on the rotation, requiring the overall rotationof the eyeball not exceeding 45°, namely, (sin² u+sin² v)⁻²<sin 45°. Onsuch a basis, the sampling is conducted once every 5° regarding u and v,and as to each group of u and v, a group of u and v shall be abandonedif the overall rotation exceeds 45°, judged by the above equation,otherwise the group of data shall be added to a synthetic parameter setof the virtual eyeball.

Conduct synthesis of the eyeball virtual appearance, regarding eachgroup of physically feasible rotation angles u and v of the eyeball. Asmentioned above, the scheme adopts the standard sphere as anapproximation of the eyeball. Meanwhile, a standard circle is attachedto the surface of the sphere, as an approximation of the iris contour.The two approximations as a whole compose a 3D model of the eyeball. Inthe specific construction, diameters of the eyeball and the iris arerespectively marked as D_(e) and D_(i), and values thereof refer tostandard parameters of the eyeball in the anatomy, namely, 25 mm and 12mm, respectively. It needs to be added that, individual differencesamong different people lead to differences in D_(e) and D_(i). Thoughfixed parameters are adopted by the present invention during modelconstruction of the virtual eyeball appearance herein, the individualdifferences are able to be effectively dealt with by the followingalgorithm.

The virtual eyeball appearance data in the case of differentline-of-sight directions may be generated, by using the eyeball model,combining the aforementioned synthetic rotation parameters of thevirtual eyeball. Firstly, making g=[g_(x), g_(y), g_(z)]^(T) as a 3Dunit vector for describing the eyeball orientation (i.e., the 3Dline-of-sight direction), and in regard to any physically feasible u andv, a calculation method of the corresponding g is as follows:

$g = \begin{bmatrix}{\sin\; u} \\{\sin\; v} \\\begin{pmatrix}1 & {\sin^{2}u} & {\sin^{2}v}\end{pmatrix}^{- 2}\end{bmatrix}$

At the same time, 3D coordinates of the eyeball center, the iris centerand the i^(th) point on the iris contour are respectively marked asE(g), C(g), P_(i)(g), for the sake of calculating the virtual eyeballappearance under the eyeball orientation. And the relationship betweenE(g), C(g), P_(i)(g) and g may be calculated by the following equation:

${C(g)} = {{\begin{bmatrix}1 & 0 & 0 \\0 & {\cos\;\beta} & {\sin\;\beta} \\0 & {\sin\;\beta} & {\cos\;\beta}\end{bmatrix}\begin{bmatrix}{\cos\;\alpha} & 0 & {\sin\;\alpha} \\0 & 1 & 0 \\{\sin\;\alpha} & 0 & {\cos\;\alpha}\end{bmatrix}}\left( {{E(g)} + \begin{bmatrix}0 \\0 \\{D_{e}/2}\end{bmatrix}} \right)}$ ${P_{i}(g)} = {{{\begin{bmatrix}1 & 0 & 0 \\0 & {\cos\;\beta} & {\sin\;\beta} \\0 & {\sin\;\beta} & {\cos\;\beta}\end{bmatrix}\begin{bmatrix}{\cos\;\alpha} & 0 & {\sin\;\alpha} \\0 & 1 & 0 \\{\sin\;\alpha} & 0 & {\cos\;\alpha}\end{bmatrix}}{E(g)}} + \begin{matrix}{{D_{i}/2}\mspace{25mu}\sin\;\gamma} \\{{D_{i}/2}\mspace{25mu}\cos\;\gamma} \\\left( {\frac{D_{e}^{2}}{4}\mspace{25mu}\frac{D_{i}^{2}}{4}} \right)^{- 2}\end{matrix}}$

Where,

${\alpha = {\arctan\left( \frac{g_{x}}{g_{z}} \right)}},{\beta = {\arctan\left( \frac{g_{y}}{g_{z}} \right)}},$and g_(x), g_(y) and g_(z) are respectively components of the aboveline-of-sight direction g along three coordinate axes. γ is azimuth ofthe i^(th) point on the iris contour relative to the iris center. Thus,regarding any physically feasible u and v, under the given 3Dcoordinates E(g) of the eyeball center, the 3D coordinates of thecorresponding eyeball orientation g, the iris center C(g) and the iriscontour {P_(i)(g)} have all be obtained via calculation. 3D coordinatesfor describing the virtual eyeball appearance under a series ofdifferent eyeball orientations, may be calculated, via traversing allthe physically feasible eyeball rotation angles u and v.

Finally, in order to synthesize the virtual eyeball appearance of a 2Dimage, 3D coordinates need to be transform into 2D pixel coordinates onthe image plane. And the following calculation is conducted using astandard camera imaging equation:

${z\begin{bmatrix}p \\1\end{bmatrix}} = {{K\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & 1 & 0\end{bmatrix}}\begin{bmatrix}P \\1\end{bmatrix}}$

Wherein, the capital letter P generally refers to any known 3Dcoordinates, and may be the aforementioned E(g), C(g), P_(i)(g). Thesmall letter p refers to 2D pixel coordinates on the correspondingimage. K is an intrinsic matrix of the camera, and may be calibrated viaexperiment, or be specified according to the real situation. z is aproportionality constant generated in the process of calculation.

In some cases, when the intrinsic reference of the camera, K, is notable to be determined, and the 3D coordinates of the eyeball center arenot specified, the following approximate method may be adopted tocalculate the synthetic virtual eyeball appearance. Firstly, set e(g) as0, and then use a simple transformation equation:

$p = {{s\begin{bmatrix}1 & 0 & 0 \\0 & 1 & 0\end{bmatrix}}P}$

Herein, s is a simple scaling constant, which can be obtained by roughlyestimating the iris diameter in the human eye image, and dividing theiris diameter by D_(i).

In the above implementation process, it is realized that, for anyphysically feasible eyeball orientation (the line-of-sight direction)g_(n), the 3D eyeball data is calculated, and is synthesized to obtain2D pixel coordinates, after such processes, specifically, 2D imagecoordinates data of the iris center c(g_(n)), and the iris contour pointset {p_(i)(g_(n))} are obtained, in which, {p_(i)(g_(n))} represents aset composed of corresponding p_(i)(g_(n)) after all the values aretraversed by i. And all the g_(n) and the corresponding c(g_(n)), and{p_(i)(g_(n))} are saved, so as to obtain the synthetic dataset of thevirtual eyeball appearances under different line-of-sight direction.

Referring to FIG. 3, which is a diagram illustrating the matching of ahuman eye image with virtual eyeball appearance according to the presentinvention, in which the synthetic dataset of virtual eyeball appearancesunder different line-of-sight direction, as described above, is used,for eyeball appearance matching with actually shot human eye images, soas to realize query and calculation of the eyeball orientation, namely,the 3D line-of-sight direction. And the specific implementation methodis as follows.

Firstly, pretreatment to the actually shot human eye image is requiredbefore the line-of-light calculation. The pretreatment is conducted fortwo purposes: brightness correction of the human eye image shot underdifferent conditions, as well as extraction of credible iris contourpixels from the human eye image. Concerning the brightness correction,firstly, histogram adjustment to the image brightness is conducted, soas to enhance the contrast ratio of the brighter region (such as thesclera) against the darker region (such as the iris and the cornea), andis implemented by the following operations:

$I_{k}^{\prime} = \left\{ {{\begin{matrix}I_{k} & {{{if}\mspace{14mu} I_{k}} < {{median}\left( I_{k} \right)}} \\{{median}\left( I_{k} \right)} & {{the}\mspace{14mu}{others}}\end{matrix}I_{k}^{''}} = {255 \times \frac{I_{k}^{\prime}\mspace{20mu}{\min_{k}\left( I_{k}^{\prime} \right)}}{\max_{k}{\left( I_{k}^{\prime} \right)\mspace{20mu}{\min_{k}\left( I_{k}^{\prime} \right)}}}}} \right.$

Wherein, I′_(k) and I″_(k) are values of each pixel point in the imagebefore and after the histogram adjustment, median( ) is a function forcalculating the median. The extraction of the iris contour from theimage after the brightness correction is specifically as follows:recording brightness variation range of the pixels in the image, andselecting the darkest region with a magnification of 0.8 as thethreshold; selecting the region with the largest area, and enhancing theregion by an image expansion operation, to obtain an approximate rangeof the iris region, and marking the central position of the region as o.Meanwhile, calculating one orientation mask from the original image, andas to the calculation method, for each pixel k, the mask value M_(k) isas follows:

$M_{k} = \left\{ \begin{matrix}1 & \begin{matrix}{{if}\mspace{14mu}{angle}\mspace{14mu}{between}\mspace{14mu}{gradient}\mspace{14mu}{of}\mspace{14mu} I_{k}\mspace{14mu}{and}} \\{o\mspace{14mu}{indicative}\mspace{14mu}{pixel}\mspace{14mu} k\mspace{14mu}{is}\mspace{14mu}{less}\mspace{14mu}{than}\mspace{14mu} 30{^\circ}}\end{matrix} \\0 & {otherwise}\end{matrix} \right.$

Analyze correlation between the mask and pixel points in the irisregion. It can be concluded from observation that, gradient direction ofthe pixels on the iris contour from the darker iris region to the brightsclera region shall depart from the center o of the iris. Therefore, asfar as the pixel k in the iris region is concerned, if the mask valueM_(k) of the point k is 1, then it indicates that the point is apotential pixel point on the iris contour. In addition, a possible iriscontour region is constrained between two sector regions extending tothe left and the right, with o as the center, so as to avoid theinfluence of the edges of the upper and lower eyelids. Finally, obtainthe possible results of the iris contour. The incompleteness of the maskmay lead to the existence of isolated pixels, therefore, determineconnectivity of all pixels, and only reserve the non-isolated pixels,resulting in the final pixel collection of the iris contour.

Secondly, select the synthetic appearance data under a certain eyeballorientation g_(n) from the virtual eyeball appearance dataset, and matchthe synthetic appearance data with the human eye image after brightnesscorrection and the iris edge image. The matching is conducted inaccordance with the two following principles:

1) circular symmetry measuring: measuring the circular symmetry of thepixel gradient of the human eye image in the iris contour region. Thecalculation method is carried out by representing the image with polarcoordinates. In the representation with the polar coordinates, thehorizontal axis represents the azimuth of the pixel point p_(i)(g_(n))on the iris relative to the iris center c(g_(n)), and the vertical axisrepresents Euclidean coordinates of the line connecting the iris centerc(g_(n)) and the pixel point p_(i)(g_(n)) on the iris. On such basis,the following method is adopted to collect data. Collecting a group ofcontinuous pixel values across the iris contour, in the direction fromc(g_(n)) to p_(i)(g_(n)) on the human eye image, in which the range ofthe collected region is proportional to the distance between c(g_(n))and p_(i)(g_(n)). Filling the pixel values into the poplar coordinates,in which the horizontal coordinates are also the azimuths ofp_(i)(g_(n)), and the vertical coordinates are relative coordinates ofthe group of the pixels after arrangement thereof from the inside of theiris contour outward, therefore, the filling results in a verticalcolumn. After completing the filling of each column under the polarcoordinates, calculating the vertical gradient thereof to obtain a polargradient map (a gradient matrix of the iris pixels). It could beconcluded from observation that, ideal matching results of the iris edgecan ensure structural similarity among columns of the matrix gradient,that is, values with higher strength appear in several same rows. Andfor the sake of quantitative measurement of consistency of each columnof the matrix, indicators such as a rank of the matrix may be used toconduct the calculation. For example, the following concentrationfunction may also be used to measure the similarity between the matrixcolumns:

${{RS} = \frac{\sigma_{1}}{\sum\left( \sigma_{l} \right)}},{{s.t.\mspace{14mu}{PGM}} = {{U\begin{bmatrix}\sigma_{1} & \; & \; \\\; & \sigma_{2} & \; \\\; & \; & \ldots\end{bmatrix}}V}}$

Herein, the polar gradient map (PGM) is a matrix, {σ_(l)} is a singularvalue, wherein l=1, 2, . . . , U and V are two matrixes generated afterPGM matrix is subject to singular value decomposition. A larger RS valueindicates a better matching.

2) iris contour matching: performing direct matching of the pixel set{q_(j)} on the iris contour edge extracted from the human eye image withthe iris contour {p_(i)(g_(n))} of the synthetic data of the virtualeyeball appearance. And due to the fact that the iris derived from thesynthetic data may differ from the actually shot iris in the dimension,the present invention proposes a robust contour matching technique tosolve this problem. In particular, the following matching criteria areproposed:

${ICF} = {\sum\limits_{j}{1{\sum\limits_{j}\sigma_{j}}}}$${Wherein},{\sigma_{j} = \left\{ \begin{matrix}{1,} & {{{if}\mspace{14mu}{\min_{i}\left( {{q_{j}\mspace{20mu}{p_{i}\left( g_{n} \right)}}}_{2} \right)}} < {{\min_{i,j}\left( {{q_{j}\mspace{20mu}{p_{i}\left( g_{n} \right)}}}_{2} \right)} +}} \\{0,} & {otherwise}\end{matrix} \right.}$

∥ ∥₂ represents 2-norm calculation, is a constant that tolerates smalldistance errors, and can be set as 2 pixels. The distances between theiris contour edge points in the image and the iris contour of thevirtual eyeball data are calculated, and at the same time, the number ofthe cases where the distances are different from the other distances iscounted by the above equation. Therefore, an ideal match should make allthe distances equal to each other, namely, ICF is 0, a requirement apoor match is incapable of. The larger the ICF value, the poor thematching results. And the matching criterion does not requires that thepixels on the iris edge in the human eye image completely coincide withthe iris contour of the synthetic data, only requires that the distancesat all positions are equal (be of the same shape), thus solving theproblem of different dimensions of the two cases.

In combination with the criterions under which the circular symmetrymeasurement and the iris contour matching are carried out, a method forcomprehensive measurement of the matching degree between the human eyeimage and the virtual eyeball appearance is RS λ ICF, wherein λ is setas 0.01, for balancing the weights of these two items.

Referring to FIG. 4, which is a flowchart illustrating calculation ofthe line-of-sight direction based on a single human eye image accordingto the present invention, the specific implementation process forcalculating the line-of-sight direction based on a single human eyeimage will be discussed in conjunction with the specific relativetechnology described above.

Firstly, input a single human eye image, conduct initial processing ofthe image, to obtain a human eye image with corrected brightness, andobtain a set of credible pixels on the iris contour edge from the humaneye image. After that, in order to calculate the unknown line-of-sightdirection g, specify a searching criterion and scope of theline-of-sight direction, and traverse the possible line-of-sightdirections one by one.

Assume g is given a new value, then set the relative translation vectoras b=0. Then, select data such as iris appearance corresponding to theline-of-sight direction which is the most approximate to g, from thesynthetic dataset of the virtual eyeball appearance. Calculatingscore=RS λ ICF, and conducting optimal updating of b via gradientdescent method under the criteria of a largest score value, repeatingthe step, until the value of b is stable. Finally, select the next valueof g.

When the value of g has been traversed and no more new values, outputthe eyeball orientation g and the relative translation vector bcorresponding to the largest score value during the whole calculationprocess which are the final output results, wherein the eyeballorientation g is also the 3D line-of-sight direction.

Overall, the process solves the following optimization problems:{g,b}=arg max_(g∈{g) _(n) _(},b) {RS(g,b)λICF(g,b)}

λ is a weight parameter, and is set as 0.01, g and b are theline-of-sight direction and the relative translation to be calculated.Referring to FIG. 5, which is a flow chart illustrating jointoptimization of the line-of-sight direction based on multiple human eyeimages according to the present invention, an implementation method forcalculating the joint optimization of the line-of-sight direction usingmultiple human eye images is described as below, on the basis of theaforementioned calculation method of the line-of-sight direction basedon a single human eye image. Firstly, using the same camera tocontinuously shoot multiple human eye images in a short time. It ispossible for the images to be accurately aligned by simple translationor rotation, because the posture of the head of the user tends to besteady in a short time. After that, respectively calculate line-of-sightdirection g_(m) and the coordinates e_(m) of the eyeball center of eachimage, using the above calculation method of the line-of-sight directionfor a single image.

Since rotation of the eyeball doesn't change the position of the eyeballcenter when the multiple human eye images have all be aligned, all e_(m)should correspond to the coordinates of the same real eyeball center,which is marked as e. Assume there is a total of M human eye images,then the average value thereof may be directly calculated as:

$e = {\frac{1}{M}{\sum\limits_{m = 1}^{M}e_{m}}}$

Further, in order to obtain a higher accuracy, only one credible subsetin {e_(m)} is expected to be used. Therefore, cluster all data in{e_(m)}, and select the largest subset therein to undergo weightedaveraging, so as to obtain e.

After obtaining the coordinates e of the eyeball center, conductingoptimization calculation again for each human eye image, and theoptimization method is still intended to maximize the followingfunction: score=RS λ ICF, which includes optimization criterions of bothcircular symmetry and the iris contour matching. The method differs fromthe above method in that, during the calculation of RS and ICF, thecoordinates e_(m) of the eyeball center, which would have taken anyvalues without constraint, are compulsorily fixed as e, and theoptimization calculation is carried out on such basis. And because theeyeball center e is fixed, an equivalent of binding the variables ofeyeball orientation and the relative translation vector, which areoriginally independent, significantly narrowing the solution space; andmeanwhile, conducting optimized search within a certain floating range(such as in the scope of ±15°), by using the result g_(m) generated fromindependent calculation as an initial value, does not require traversingthe entire solution space. Therefore, it is possible to obtain theoptimized line-of-sight direction ĝ_(m) quickly. Overall, the jointoptimization process solves the following problem:ĝ _(m)=arg max_(g∈{g) _(n) _(},b) {RS(g,e)λICF(g,e)}s.t. arccos(g g_(m))<15°

Wherein, the 3D line-of-sight direction ĝ_(m) is the final outputtargeted at the m^(th) image.

In conclusion, the novel optimization matching criterion of the eyeballappearance of the present invention, effectively reduces effects ofuncontrollable factors, such as image scaling and noise on the results;and a joint optimization method is invented for the case of continuouslyshooting multiple human eye images, so as to further improve thecalculation accuracy. One of the applications of the present inventionis virtual reality and human computer interaction, under the principlethat, shooting eye images of a user, and calculating the line-of-sightdirection of the user enable interaction with an intelligent systeminterface or a virtual realistic object. The present invention can alsobe widely used in training, games and entertainment, video surveillance,medical care and other fields.

What is described above is only one representative embodiment of thepresent invention, and any equivalent transformation made on the basisof the technical schemes of the present invention shall fall within theprotection scope thereof.

What is claimed is:
 1. A calculation method of line-of-sight directionbased on analysis and match of iris contour in human eye image,comprising the following steps: (1) constructing a sphere eyeball model,traversing eyeball orientations with different physical feasibilities,generating 2D virtual eyeball appearances with different orientations bygeometric calculation, and storing all the eyeball orientations andcorresponding virtual eyeball appearance data in a dataset for use inspecific applications; (2) during application, firstly shooting a facialimage of a user, fixing on a left-eye or right-eye region, pre-treatinga human eye image, completing brightness correction and extractingpixels on iris edges in the human eye image; (3) regarding the shot andpre-treated human eye image and the virtual eyeball appearance data inthe dataset, matching the human eye image with the virtual eyeballappearance data via a matching optimization algorithm of the human eyeimage and the virtual eyeball appearance data, wherein a matching resultdetermines an orientation and a position of the eyeball which best matchwith the human eye image; and (4) regarding continuously shot human eyeimages, further conducting joint optimization on the basis of theeyeball appearance matching in step (3), on the condition that a centralposition of the eyeball remains unchanged, or that the human eye imageshave been aligned, and precisely and simultaneously calculating 3Dline-of-sight direction corresponding to each image; wherein a methodfor the joint optimization in step (4), and the method for accuratelycalculating the 3D line-of-sight direction corresponding to each imagesimultaneously are as follows, under the assumption that centralposition of the eyeball remains unchanged or has been aligned when theimages shot, conducting the matching of the human eye images with thevirtual eyeball appearance in step (3), and calculating the eyeballorientation corresponding to each human eye image and the coordinates ofthe central position of the eyeball; excluding the central coordinatesof the eyeball with obvious deviation therein, and conducting weightingcalculation of standard coordinates of the eyeball center using theremaining coordinates; and individually carrying on with theoptimization in step (3), while adding one optimization constraint,namely, coinciding the matched eyeball central coordinates as much aspossible with the standard eyeball central coordinates, and updating thecalculation results as the eyeball orientations in the human eye images,namely, the final results of the 3D line-of-sight direction.
 2. Thecalculation method of line-of-sight direction based on analysis andmatch of iris contour in human eye image according to claim 1, whereinthe virtual eyeball appearance data in step (1) are generated asfollows, firstly, establishing a 3D sphere model of the eyeball, andadding important elements of a circular iris contour on the surface ofthe 3D sphere model; traversing different physically-feasible eyeballorientations, namely, different rotation angles around a horizontal axisand a vertical axis respectively, for each of the rotation angles,projecting a corresponding 3D virtual eyeball appearance to a 2D plane,namely, to a position just in front of the corresponding eyeball, andrecording projected iris contour coordinates, eyeball centralcoordinates and 2D eyeball appearance data of the iris centralcoordinates; and storing all the eyeball orientations and the virtualeyeball appearance data corresponding to the eyeball orientations in thedataset.
 3. The calculation method of line-of-sight direction based onanalysis and match of iris contour in human eye image according to claim1, wherein the matching optimization algorithm of the human eye imageand the virtual eyeball appearance data in step (3) is as follows,matching parameters to be calculated are relative translation amount ofthe human eye image and the virtual eyeball appearance in a 2D imagedomain, and the eyeball orientation corresponding to the virtual eyeballappearance, and optimizing matching degree of the human eye image andthe virtual eyeball appearance via finding the best values of the twomatching parameters, so as to realize matching of the human eye imageand the virtual eyeball appearance data.
 4. The calculation method ofline-of-sight direction based on analysis and match of iris contour inhuman eye image according to claim 3, wherein the matching degree iscalculated by the following functional measurements: (31) measurement ofcircular symmetry: pixels in the proximity of the virtual eyeball iriscontour in the human eye image have better circular symmetry, when thematching tends to become ideal; and (32) measurement of iris contourmatching degree: calculating distances between pixels on the iris edgein the human eye image and the virtual eyeball iris contour, when thematching tends to become ideal, and the distances tend to become equal.5. The calculation method of line-of-sight direction based on analysisand match of iris contour in human eye image according to claim 4,wherein the measurement of the circular symmetry in step (31) isconducted as follows, coordinates of the iris contour of the virtualeyeball appearance and coordinates of the central position of the iriscorresponding to any one group of matching parameter may be determined;overlapping these coordinates onto the human eye image, via continuouslysampling values of pixels on the human eye image, along both thepositive and negative directions indicated by lines connecting the iriscenter of the virtual eyeball appearance and points on the iris contour,while taking the points as reference, so as to obtain 1D pixel columnvectors, and the sampling region is proportional to the distancesbetween the points on the iris contour of the virtual eyeball appearanceand the iris center of the virtual eyeball appearance; sampling viatraversing the points on the iris contour of the virtual eyeballappearance, and combining all the obtained 1D column vectors into a 2Dmatrix; and finally, calculating distribution consistency of each columnof numeric of the matrix or a gradient matrix of the matrix in thevertical direction, the consistency may be measured in a matrix kernelfunction, the correlation coefficient of each column, the concentrationof singular values, and the higher the consistency, the better thecircular symmetry.
 6. The calculation method of line-of-sight directionbased on analysis and match of iris contour in human eye image accordingto claim 4, wherein the measurement of the iris contour matching degreein step (32) is conducted as follows, regarding any one group ofmatching parameters, determining coordinates of the iris contour of thevirtual eyeball appearance corresponding to the matching parameters ofthe group; traversing the pixels extracted from the iris edge of thehuman eye image, and calculating the distances between the edge pixelsand the iris contour of the virtual eyeball appearance; and reviewingdistribution of the distances, in which the less the number of thedistances which are significantly different from the others, the betterthe measurement result of the iris contour matching degree.